can we add the beacon api to the engine api tile? it seems overkill to have cores assigned to marshalling http requests and responses for each api?

we could make this a component that is not an actual Tile but has a spin or loop method that could be called from a tile? that would give some flexibility in how it is deployed.

can we add the beacon api to the engine api tile? it seems overkill to have cores assigned to marshalling http requests and responses for each api?

we could make this a component that is not an actual Tile but has a spin or loop method that could be called from a tile? that would give some flexibility in how it is deployed.

also - if we run the http server as a separate tile then we should just run Axum in a single thread runtime and not take on the maintenance risk of writing our own web-server.

can we add the beacon api to the engine api tile? it seems overkill to have cores assigned to marshalling http requests and responses for each api?

we could make this a component that is not an actual Tile but has a spin or loop method that could be called from a tile? that would give some flexibility in how it is deployed.

I do think the current tile setup is likely not optimal but we should probably wait until everything is running and we've gathered a bunch of data before deciding to consolidate tiles? When feature complete the system will have 3 integration points. p2p, Beacon api, and engine api. Currently it looks like we have 3 tiles for p2p stuff. Putting the Beacon and engine api in the same tile seems premature.

can we add the beacon api to the engine api tile? it seems overkill to have cores assigned to marshalling http requests and responses for each api?
we could make this a component that is not an actual Tile but has a spin or loop method that could be called from a tile? that would give some flexibility in how it is deployed.

also - if we run the http server as a separate tile then we should just run Axum in a single thread runtime and not take on the maintenance risk of writing our own web-server.

I don't see why we'd go in that direction. That would pull in all of tokio and so on. Seems like the opposite to our approach elsewhere. The Beacon api is the interface to the validator client and commit boost and so on. If our design goals center low level, high performance, and innovation we probably shouldn't outsource a key interface to the axum, hypr, tokio stack.

Axum is just a nice wrapper around hypr and hypr is just state machine that sits on the same two crates I'm using here. Mio and httparse. And unlike the p2p interface, this one is purely internal so minimal security risk.

can we add the beacon api to the engine api tile? it seems overkill to have cores assigned to marshalling http requests and responses for each api?
we could make this a component that is not an actual Tile but has a spin or loop method that could be called from a tile? that would give some flexibility in how it is deployed.

I do think the current tile setup is likely not optimal but we should probably wait until everything is running and we've gathered a bunch of data before deciding to consolidate tiles? When feature complete the system will have 3 integration points. p2p, Beacon api, and engine api. Currently it looks like we have 3 tiles for p2p stuff. Putting the Beacon and engine api in the same tile seems premature.

well yes to not predetermining the tiles - which is why it makes sense as much as possible to create components that are not tiles but can be called from tiles

can we add the beacon api to the engine api tile? it seems overkill to have cores assigned to marshalling http requests and responses for each api?
we could make this a component that is not an actual Tile but has a spin or loop method that could be called from a tile? that would give some flexibility in how it is deployed.

also - if we run the http server as a separate tile then we should just run Axum in a single thread runtime and not take on the maintenance risk of writing our own web-server.

I don't see why we'd go in that direction. That would pull in all of tokio and so on. Seems like the opposite to our approach elsewhere. The Beacon api is the interface to the validator client and commit boost and so on. If our design goals center low level, high performance, and innovation we probably shouldn't outsource a key interface to the axum, hypr, tokio stack.

Axum is just a nice wrapper around hypr and hypr is just state machine that sits on the same two crates I'm using here. Mio and httparse. And unlike the p2p interface, this one is purely internal so minimal security risk.

the concern is not security, its just maintaining code which implements something that is already available and widely used - I see no issue in running tokio in a separate tile - or separate process on the same machine - if its for non-hot-path things.

I have worked with Claude.ai to prepare a new design for Beacon API - the basic principles have been added to this branch in d096b51. Comments below will provide more details.

Design notes from Claude.ai

Goal (from team, channeled by Bronek)

• No separate tile (loop_body) for any API access — serving (beacon_api) or calling (engine).
• One new spine-attached tile (working name client_server) hosting hardcoded crates: beacon_api (kept name, refactored mechanics, HTTP server) and engine_api (renamed from engine, HTTP client).
• Open: seam location for HTTP machinery (host implements client+server and provides them to hosted crates, vs crates implement internally).
• Wanted: table-based dispatch for HTTP request handling (server side). Open: same for engine_api client side — decide on pros/cons.
• Principles: simplicity without performance compromise; for HTTP specifically, simplicity + consistency take precedence (unavoidable multi-ms latency tax).

Mapped facts the designs rest on

• TileConfig::new(n, None): n is a CPU core id; each tile = one OS thread pinned to its own core, busy-polling. Cores 1–7 taken (Controller, Network, BeaconState, Storage, Engine, DataColumns, BeaconApi). Merging engine + beacon_api frees a core — the mechanical driver behind "one tile for APIs".
• crates/beacon_api (prototype, 596 lines): mio+httparse server, hardcoded 0.0.0.0:5051, blocking poll(100ms) in loop_body (must die in a shared loop), exact-match routing via closure F: Fn(&ParsedRequest, &mut Vec<u8>), 2 endpoints, no live state, 16 MiB/conn read buffers, no conn cap, ignores spine adapter. In-code TODO: parameterised routing needed.
• crates/engine (mature): non-blocking mio client (poll Duration::ZERO), HttpPool one-req-per-conn growing unboundedly (10 MB read + 10 MB write per conn), hand-rolled JWT HS256 (1 s token cache), simd_json. One free function per method; response correlation FxHashMap<u64, ReqKind>; dispatch = match on EngineReq (inbound) and ReqKind (completion). HOT PATH: newPayload transcodes SSZ→JSON straight into persistent 10 MB scratch (client.rs:156-177, types.rs:269,345) — must be preserved verbatim. Dead parallel UDS transport ipc.rs (474-line near-copy of http.rs).
• Spine contract (unchanged by any design): queues engine_reqs / engine_resps / engine_health; requests carry TCacheRead handles; responses written to tile-owned TCache. Producers: BeaconState, DataColumns.
• State access for future endpoints: BeaconStateReader::read(|rv| ..) seqlock; closures short + re-runnable; None pre-bootstrap (docs/beacon-state-architecture.md §4-5).
• Test gap: neither mio event loop has any end-to-end test (test-or-broken rule).
• SpineAdapter::connect_tile is public — tests can build a real spine and drive loop_body by hand (verified by design agent A).

Recommendation (Claude's, for team review)

1. Crates: client_server (tile: single mio Poll, token slab, loop composition), httpcore (one connection state machine, both roles, byte interfaces — no Poll, no sockets in signatures), beacon_api (ROUTES table + handlers + Request/Response/ApiCtx, transport-free), engine_api (renamed engine: codecs, correlation, JWT, ReqKind match, transport-free; scratch handed in, hot path verbatim).
2. No traits at host↔hosted seams: two hardcoded crates = plain fields + fn calls (one adapter = hypothetical seam). A future third crate edits the host tile — accepted, cheap, honest.
3. Transport testability via byte-level interfaces (C/D convergence): httpcore machines tested on byte slices with deterministic chunking (partial reads, pipelining, keep-alive, oversize) — no Mux trait needed when the seam is already &[u8].
   3a. Two production transports (team decision): TCP + UDS. Because the transport set is closed and we control it, represent it as an enum, not a trait — enum Stream { Tcp(mio::net::TcpStream), Uds(mio::net::UnixStream) } with match-forwarding read/write/register (both impl Read+Write+Source), plus enum Bind { Tcp(SocketAddr), Unix(PathBuf) } in config (B's shape). Same principle as client dispatch: closed set → enum; open set → table. Monomorphic, no generics in hosted-crate signatures (D's TcpOrUds). UDS applies to the engine_api client pool first; the server side gets the same Bind enum for free. UnixStream::pair() doubles as a portless, deterministic test transport for full-machine tests (B's suggestion).
4. Dispatch: server table as converged; client enum match as unanimous.
5. Test plan (test-or-broken), three tiers:
   a. Unit (socket-free, fast): router+handler tests through the table with real seqlock fixture pairs (typos in patterns fail in tests, not prod); httpcore byte-machine tests with deterministic chunking; golden-byte engine tests incl. counting-allocator assertion that newPayload allocates zero (the hot-path invariant finally gets a failing test).
   b. Integration over REAL http://localhost (team-requested, 2026-08-14): cargo tests/ dir tests that construct the real tile (real mio Poll, real TCP listener) and hand-crank loop_body in a plain test loop — no flux threads/pinning needed since loop_body is a plain fn and SpineAdapter::connect_tile is public. Test issues real HTTP over loopback (std::net or minimal client). DESIGN REQUIREMENTS this imposes: (i) bind address configurable (kills hardcoded 0.0.0.0:5051), (ii) bind to port 0 in tests and EXPOSE the actual bound address (local_addr()) so tests discover the ephemeral port, (iii) everything non-blocking means the test loop spins deterministically — fast, no sleeps. Same technique covers the UDS bind (tempdir socket path).
   c. Full-loop: beacon-api request served over loopback WHILE an EL call is in flight against a canned local EL — the merged-loop interleaving invariant no prototype ever tested; one such smoke test per transport as the seatbelt.
6. Bounded client pool + server conn cap + configurable bind (kills three prototype liabilities in passing).

Implementation plan from Claude.ai

Commit sequence

Strategy: build the new transport alongside the old, migrate consumers one by one, consolidate tiles last. No commit mixes a move with a behavior change.

C1 — crates/httpcore skeleton (server role). b977f93
New crate silver_httpcore: the server-side connection byte-machine extracted from beacon_api's tile.rs (parse → dispatch-closure → frame response; keep-alive + pipelining preserved). Interfaces are bytes/ frames — no Poll, no sockets in signatures. Migrate beacon_api's 14 parser tests + add deterministic-chunking tests (partial reads, pipelined split across pumps, oversize reject). beacon_api crate switches to httpcore for framing but its tile stays attached and behaviorally identical (still its own Poll; the 100 ms block dies in C5, not here).

C2 — httpcore client role; engine consumes it; delete ipc.rs. 42d86da
Client-side connection machine (one-request-per-conn, Content-Length framing, WouldBlock resumption) moves from engine http.rs into httpcore, generic over Stream. ipc.rs (474 dead lines) deleted — UDS is now a Stream variant. Engine crate keeps: pool policy, JSON-RPC, JWT, correlation, ReqKind — all protocol. Engine's 66 tests keep passing; add golden-byte round-trip test through the new machine incl. JWT header. Bounded pool cap introduced here (config, default generous).

C3 — beacon_api table dispatch + Request/Response/ApiCtx. e7cbd99
Const ROUTES table ((Method, pattern) → handler fn), pattern compiled to Lit/Param segments at init, zero-alloc Params (inline ≤4), router owns 404/405, handlers own 400; ApiCtx { state: BeaconStateReader, identity, outbox }. Rewrite the two existing endpoints as table rows; /metrics still stubbed. Router+handler tests through the table with fixture seqlock pairs; 503-before-bootstrap test. [AMENDED: Accept/content-type field deferred to the M2 SSZ-negotiation work — httpcore doesn't expose request headers yet, and a parsed-but-unread field is dead API. Outbox/NodeCommand deferred to its first consumer (same principle as the C1 Stream/Bind deferral). Reader-in-ApiCtx conditional on cheap fixture + srv.rs-example construction — implementer investigates and reports. New behavior sanctioned: 405 for known-path-wrong-method (today the method is ignored entirely).]

C4 — rename crates/engine → crates/engine_api. 96a900c
Purely mechanical, own commit: directory, package name silver_engine_api, workspace member + dep entries, imports, docs/spine-message-flow.md prose. "Names track current reality" — also rename lingering engine-named locals in touched files. No logic changes.

C5 — crates/client_server tile; consolidation. 36f291e
New tile crate constructs/holds the beacon_api server component and the engine_api client component; loop_body composes their pumps (all non-blocking). [AMENDED: TWO Polls, not the single shared Poll + token slab from the recommendation sketch — C2 landed the pool with its own Poll inside engine_api's verified event paths; unifying would churn them to save one epoll_wait(0)/iteration (noise per team latency ruling) at the cost of token-space partitioning. Design D's argument, now backed by C2's shape. Revisit only if measurement ever says otherwise. No outbox — deferred since C3.] beacon_api's own tile and EngineTile deleted; main.rs attaches client_server once (cores renumber, one core freed); srv.rs example updated. Config: beacon-api bind via Config builder + file + --beacon-api-bind; execution_endpoint accepts socket path; tile exposes local_addr(). Tests: tier-b integration (real localhost HTTP against hand-cranked loop_body via public SpineAdapter::connect_tile, port 0 + local_addr); tier-c full-loop (beacon-api request served WHILE canned-EL call in flight); UDS variants of both (tempdir socket path).

C6 — counting-allocator hot-path test + smoke. bd42255
Zero-allocation assertion on the newPayload transcode path (CountingAllocator exists in main.rs idiom); one real-socket smoke per transport per direction. Any leftover prototype liabilities: connection cap on server, 16 MiB→sane read cap, /eth/v1/events → clean 404.

@vladimir-ea @0w3n-d note a limitation in the C5, we may want to revisit it:

Single beacon_api bind As of C5 the node binds exactly ONE listener — TCP or UDS, chosen by beacon_api_bind. If we want simultaneous surfaces (local VC over UDS + TCP on specific interfaces), the additive change is: BeaconApi holds a list of listeners (one reserved token each; accept loop per listener; connections already transport-uniform via Stream), config becomes a list of binds. Router, handlers, connection handling untouched.

Recommendation (Claude's, for team review)

1. Crates: client_server (tile: single mio Poll, token slab, loop composition), httpcore (one connection state machine, both roles, byte interfaces — no Poll, no sockets in signatures), beacon_api (ROUTES table + handlers + Request/Response/ApiCtx, transport-free), engine_api (renamed engine: codecs, correlation, JWT, ReqKind match, transport-free; scratch handed in, hot path verbatim).

agree that a single mio::Poll should be used across both apis - note that you can have a blocking poll if we are using the Flux thread_park feature (which we should probably make the default for silver?) b/c that hooks the Poll into the signalling mechanism (via mio::Waker https://github.com/gattaca-com/flux/blob/c73a663f62111fd4466551e53790b2d2f576608f/crates/flux-communication/src/park.rs#L160).

@vladimir-ea @0w3n-d note a limitation in the C5, we may want to revisit it:

Single beacon_api bind As of C5 the node binds exactly ONE listener — TCP or UDS, chosen by beacon_api_bind. If we want simultaneous surfaces (local VC over UDS + TCP on specific interfaces), the additive change is: BeaconApi holds a list of listeners (one reserved token each; accept loop per listener; connections already transport-uniform via Stream), config becomes a list of binds. Router, handlers, connection handling untouched.

Probably OK for now. UDS would most likely be used for out bound connections to the EL, as EL and CL often run on the same machine. But on the listener side normally people run a VC on a different machine for security, maybe used by solo stakers or for testing but TCP is probably fine for those cases too.

Proposed API scope by Claude.ai

Beacon API: the validator-client surface (scoping beacon_api v1)

Date: 2026-08-14. Method: primary sources only — ethereum/beacon-APIs OpenAPI at tag v4.0.0 (current stable, 2025-10-14, Fulu-era; v5.0.0-alpha.x = Gloas dev), plus source of five VCs: Lighthouse sigp/lighthouse@unstable (LH), Teku Consensys/teku@master (TK), Nimbus status-im/nimbus-eth2@unstable (NB), Prysm OffchainLabs/prysm@develop (PR), Vouch attestantio/vouch + attestantio/go-eth2-client (VO). Five parallel research agents, one per client; every claim below carries a file path.

Caveat: PR's REST mode is still experimental and flag-gated (--enable-beacon-rest-api, config/features/flags.go:128; gRPC stays default through v8). PR-only rows are weighted accordingly.

Spec baseline (v4.0.0): v1 variants of pool attestations, aggregate_attestation, aggregate_and_proofs, and block/blinded-block publish are REMOVED (release notes, PR #549); block production is v3-only. So the Electra/Fulu-era surface is v2/v3 where marked below — silver never needs the v1 variants. Gloas (v5-alpha) later adds: v4 blocks, v2 proposer duties, ptc duties, payload_attestation_data, pool/payload_attestations, execution_payload_envelopes/bids, proposer_preferences, v2 node/version, topics head_v2 / execution_payload_available. LH/TK/PR already carry client code for these — out of v1 scope but the routing table should expect them.

(a) Endpoint table

Spec link = path under github.com/ethereum/beacon-APIs/blob/v4.0.0/. Verdict: MUST = exercised by ≥1 surveyed VC in default-config normal operation; SHOULD = specific clients / common configs; OPT = niche, subcommand, or DVT.

MUST — core duty cycle (all five VCs unless noted)

SHOULD — client- or config-specific

OPT — niche / out of normal operation

(b) /eth/v1/events per client — NO client hard-requires it

• LH (new on unstable): head monitor ON by default (enable_beacon_head_monitor: true, validator_client/src/config.rs L139), topic head only (beacon_node_fallback/src/beacon_head_monitor.rs L109). Purpose: attest immediately on head arrival. Stream failure = warn + auto-restart; attestations fall back to the ~1/3-slot deadline poll (attestation_service.rs L297–327). --disable-beacon-head-monitor exists.
• TK: EventSourceBeaconChainEventAdapter subscribes head always; + attester_slashing, proposer_slashing only with --shut-down-when-validator-slashed-enabled (default false). ForkAwareTimeBasedEventAdapter runs CONCURRENTLY from genesis — all duty timers fire with or without the stream; SSE only adds early attestation + reorg-aware duty refresh.
• NB: block monitor --block-monitor-type ∈ {disabled, poll, event}, default event (conf.nim:1113), topic head only (block_service.nim:503). poll mode (3× GET headers/head per slot) is a fully supported alternative; SSE failure = debug log + retry.
• PR: topics head_v2 + execution_payload_available (api/client/event/event_stream.go:34); auto-falls back to head if the BN 400s on head_v2 (multi_event_stream.go L145). Reconnects forever (1 s→16 s backoff); duties are polled regardless. Without events: attests at full deadline, no dependent-root duty refresh.
• VO: topics head + block (services/controller/standard/service.go:187-188), SSE via r3labs/sse, reconnect every 1 s (gec http/events.go:62). Drives early attest/sync-msg (fastTrackJobs), reorg duty re-fetch, block-root cache. Attestation jobs are ALSO scheduled at slot+slotDuration/3 (attester.go:110) — stream absence slows, not breaks.

Conclusion: polling fallback exists everywhere; the penalty for omitting SSE is degraded attestation latency (no early attest), no reorg-triggered duty refresh, and constant reconnect hammering (VO every 1 s, PR ≤16 s, LH/NB retries) polluting logs on both sides. Union of topics to serve when implemented: head, block (head_v2 declined via 400 is handled by PR). Slashing topics only for TK's opt-in shutdown feature.

(c) SSZ vs JSON

MUST accept/serve SSZ (application/octet-stream):

• POST /eth/v2/beacon/blocks + blinded: LH sends SSZ ONLY — no JSON fallback (post_beacon_blocks_v2_ssz, eth2 lib L525; block_service.rs L741–770). TK/PR/VO also post SSZ-first (TK sticky-flips to JSON on 415; PR caches 415 per host w/ TTL; VO submitproposal.go:80). NB posts JSON. => server must decode BOTH, keyed by Content-Type + Eth-Consensus-Version request header.
• GET /eth/v3/validator/blocks/{slot}: all five send Accept preferring octet-stream (TK q=0.9 default-on flag; LH SSZ-first w/ JSON retry; NB preferSSZ; PR q=0.95; VO q=1.0). Serving JSON is legal (clients switch on response Content-Type) but SSZ is the expected fast path; version/blinded-ness MUST come via response headers either way.

JSON-only in practice (spec allows SSZ on some, no surveyed VC uses it): attestation_data, aggregate_attestation v2, aggregate_and_proofs v2, pool attestations v2 (NB batch mode --batch-attestations posts SSZ, default off), register_validator (TK hardcodes JSON, OkHttpValidatorTypeDefClient), all duties, subscriptions, sync-committee family, validators queries, liveness, all bootstrap. VO alone sends SSZ-preferring Accept on GET v2 blocks/{id} (gec http.go:270) — JSON response is accepted.

v1 rule of thumb: SSZ decode on block publish, SSZ encode on block produce; JSON everywhere (incl. those two, for NB publish and any 415/downgrade path). Everything else JSON-only is spec-conformant and matches observed client behaviour.

(d) Conclusion for the client_server tile design

1. v1 must serve ~25 routes (MUST rows): 3 duties, attestation_data + 3 attestation submit/aggregate routes (v2), produce v3 + publish v2, 3 sync-committee routes + contribution pair, 2 subnet subscriptions, prepare_beacon_proposer, liveness, validators lookup (POST+GET+by-id), genesis/spec/fork_schedule/version/syncing, blocks/{id}/root, headers/{id}. The SHOULD tier adds ~10 mostly-trivial reads. Fits the const ROUTES table + linear scan design comfortably.
2. SSE is avoidable at first: every surveyed VC degrades to polling. Acceptable for bring-up; not acceptable for production (latency + reconnect spam). It slots into the accepted design exactly as anticipated: an additive subscription mode on the connection state machine (long-lived conn, small appended writes on head/block from spine events, periodic : keep-alive comments), NOT big-body streaming. Return 400 for unsupported topics per spec (PR relies on this for head_v2 negotiation).
3. Nothing else conflicts with the materialized-in-buffer model. Zero long-polling or chunked responses in any client path. Bounded worst cases: produceBlockV3 SSZ (single- digit MB), validators queries (VCs always pass their own id lists; only TK's GET fallback and VO's no-ids call could ask for more — cap/paginate is a policy choice, not a streaming need). Client call timeouts are tight (TK 10 s hard; attestation_data effectively sub-second useful window) — the sync-handler + short-seqlock-read model is the right fit.
4. Content negotiation is part of the router contract: Accept parsing on produce-v3 (and optionally attestation_data/aggregate later), Content-Type + Eth-Consensus-Version on the three v2 publish routes, broadcast_validation query on block publish, and the four produce-v3 response headers. 415 responses are load-bearing (TK/PR downgrade logic); so is 400-on-bad-topic for /eth/v1/events.

@vladimir-ea @0w3n-d note a limitation in the C5, we may want to revisit it:
Single beacon_api bind As of C5 the node binds exactly ONE listener — TCP or UDS, chosen by beacon_api_bind. If we want simultaneous surfaces (local VC over UDS + TCP on specific interfaces), the additive change is: BeaconApi holds a list of listeners (one reserved token each; accept loop per listener; connections already transport-uniform via Stream), config becomes a list of binds. Router, handlers, connection handling untouched.

Probably OK for now. UDS would most likely be used for out bound connections to the EL, as EL and CL often run on the same machine. But on the listener side normally people run a VC on a different machine for security, maybe used by solo stakers or for testing but TCP is probably fine for those cases too.

note that you do not need an 'accept loop' for each listener - it is simply another token registered with the poll and you handle as an accept when it becomes readable - this is how the tcp accept currently works with mio.

@vladimir-ea @0w3n-d note a limitation in the C5, we may want to revisit it:
Single beacon_api bind As of C5 the node binds exactly ONE listener — TCP or UDS, chosen by beacon_api_bind. If we want simultaneous surfaces (local VC over UDS + TCP on specific interfaces), the additive change is: BeaconApi holds a list of listeners (one reserved token each; accept loop per listener; connections already transport-uniform via Stream), config becomes a list of binds. Router, handlers, connection handling untouched.

Probably OK for now. UDS would most likely be used for out bound connections to the EL, as EL and CL often run on the same machine. But on the listener side normally people run a VC on a different machine for security, maybe used by solo stakers or for testing but TCP is probably fine for those cases too.

note that you do not need an 'accept loop' for each listener - it is simply another token registered with the poll and you handle as an accept when it becomes readable - this is how the tcp accept currently works with mio.

Yes, which is why I think this should be a cheap fix, so perhaps we just need to do it.

Updated docs/adr in 828ebdb to include streaming in scope, for future implementation of /eth/v1/events (polling is wasteful on resources)